NSVQ: Noise Substitution in Vector Quantization for Machine Learning

Machine learning algorithms have been shown to be highly effective in solving optimization problems a wide range of applications. Such typically use gradient descent with backpropagation and the chain rule. Hence, fails if intermediate gradients are zero for some functions computational graph, because it causes collapse when multiplying zero. Vector quantization is one those challenging machine algorithms, since piece-wise constant function its almost everywhere. A typical solution apply straight through estimator which simply copies over vector backpropagation. Other solutions based on smooth or stochastic approximation. This study proposes technique called NSVQ, approximates behavior by substituting multiplicative noise so that can used problems. Specifically, error replaced product original normalized vector, samples drawn from zero-mean, unit-variance normal distribution. We test our proposed NSVQ three scenarios various types Based experiments, achieves more accuracy faster convergence comparison estimator, exponential moving averages, MiniBatchKmeans approaches.